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ABSTRACT 



This paper reports on the development and implementation of 
a flexible audit trail system comprising a library of auditing functions that 
can be embedded into interactive courseware and customized to the 
requirements of researchers and developers. A series of essential criteria 
considered critical to the development of a robust, flexible audit trail 
system were devised. The six criteria established were: flexibility, 
efficiency, scalability, portability, retrospective compatibility, and ease 
of use. The paper presents a case study involving the pilot implementation of 
the system within a piece of medical courseware. The results of this study 
demonstrate the general utility of evaluation based on audit trails. Results 
also provided the courseware's developers with valuable information on how 
students used the courseware generally and, more specifically, on how they 
completed one particular interactive task. (AEF) 
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Abstract 

This paper reports on the development and implementation of a flexible audit trail system 
comprising a library of auditing functions that can be embedded into interactive courseware 
and customised to the requirements of researchers and developers. It presents a case study 
involving the pilot implementation of the system within a piece of medical courseware. The 
results of this study demonstrate the general utility of evaluation based on audit trails. They 
also provided the courseware’s developers with valuable information on how students used 
the courseware generally and, more specifically, on how they completed one particular 
interactive task. 



Audit Trails: One Component of Electronic Evaluation 

We are currently developing a system of electronic evaluation that addresses three distinct types of evaluation: 
(i) students’ activities and movements within individual pieces of courseware (ii) students’ use of and 
movements between pieces of courseware, and (iii) students’ perceptions of pieces of courseware. The first of 
these components, based on audit trails, provides developers, evaluators and researchers in the area of 
educational technology with an invaluable mechanism to record and analyse students' activities within a variety 
of electronic environments. Misanchuk and Schwier (1992) define and ascribe four primary roles to audit trails: 

• Formative evaluation of instructional design; that is, evaluation aimed at optimising the performance of 
a piece of courseware. 

• Basic research in instructional design; for example, investigating how different users interact with a 
piece of courseware. 

• Usage audits for unstructured or public environments; that is, determining which paths or components 
of a piece of courseware are most visited by users. 

• Counselling and advising; that is, supporting the user’s decision making within a piece of courseware. 

With these four roles in mind we devised a series of essential criteria that we considered critical to the 
development of a robust, flexible audit trail system. The six criteria we established were: 

• Flexibility - the system should be capable of fulfilling all of the roles of audit trails outlined by 
Misanchuk and Schwier (1992). It should be able to store a range of data types. 

• Efficiency - the system should manage data efficiently. It should only record and deliver information 
specifically requested by the developer/researcher and any information collected should be clearly 
associated with a specific user action or activity. 

• Scalability - the system should be capable of handling an unlimited number of records and of 
dynamically adding or deleting records. 

• Portability - the system should adopt an architecture that can be implemented across a range of 
authoring environments/software. 

• Retrospective compatibility - the system should be easily fitted to existing courseware. 

• Ease of use - the system should be easily implemented and require minimal programming effort by 
developers/researchers. 

The first two of these criteria are general and address the various roles of audit trails and the need to provide a 
robust yet flexible framework and to avoid indiscriminate data collection. The third and fourth criteria are 
essentially technical requirements that speak to the need to develop a robust yet generic audit trail system. The 
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final two criteria are concerned with adoption and implementation; ensuring the system is both useful to and 
useable by the developer/researcher. 

We subsequently designed an audit trail system that sought to address each of the above criteria. As most of the 
courseware produced within our unit is created using Macromedia Director® we elected to initially develop our 
system in and for that authoring environment. However, in keeping with our fourth design criteria - portability - 
we ensured that the system’s architecture was structured in such a way so as to be readily adapted to a range of 
other scriptable multimedia authoring systems. 

The resulting system is built around a library of functions that read and write individual records of user activities 
to a master record (the audit trail) within the target courseware. These functions interact with the other 
components of the system, which include client- and server-side administrative applications for transmitting and 
storing the audit trail data for later analysis. The library consists of a range of functions including timers, 
counters, item selection and data/text entry from which developers and researchers select according to their 
needs. Individual functions are attached to or associated with each of the objects to be audited within a piece of 
courseware and, when activated, these functions report to and dynamically update the audit trail record. The 
audit trail record can be exported at any time, although this process typically occurs when the user quits the 
courseware. 

As a pilot test of the system, we embedded it into a piece of biomedical courseware called Medical Genetix. The 
remainder of this paper describes the structure and use of the Medical Genetix courseware, our implementation 
of the audit trail system within it, and the various findings that emerged from this implementation. 



The Structure and Use of Medical Genetix 

Medical Genetix is used by medical students at the University of Melbourne to investigate biomedical and 
clinical aspects of genetic disorders such as cystic fibrosis and thallasaemia. Specifically, it is used as a self- 
directed learning resource to support students’ investigation of particular medical problems or ‘problems of the 
week’ (Keppell, Elliott, & Harris, 1998; Keppell, Kennedy & Harris, 2000). Medical Genetix is available for 
students’ use in the faculty’s main computer labs as well as in tutorial rooms. 

Medical Genetix consists of a number of sections and levels that students can easily move between. For 
example, material on the disorder cystic fibrosis is divided into three primary sections: Clinical Diagnosis, 
Laboratory Diagnosis and, Counselling and Ethics. The Clinical Diagnosis section is further broken down into 
Clinical Features, Family Histories and Pedigrees, and Molecular Pathogenesis. Each of these subsections has a 
series of screens comprising content and interactive tasks that allow students to review material, perform 
construction tasks, test their knowledge and understanding, or access further information as they require 
(Metcalfe, Williamson & Bonollo, 1999). 

Medical Genetix was made available to second-year medical students to support their investigation of a problem 
of the week on cystic fibrosis. In the Family Histories and Pedigrees section of the courseware, students can 
access an interactive task involving the construction of a pedigree on the basis of a supplied family history. The 
family history details the relationship between family members and their status with regard to the genetic 
disorder cystic fibrosis (that is, whether family members were carriers, a proband, or had the disorder). The 
interactive pedigree is completed by dragging “tiles” designating these various states into the correct position in 
the pedigree on the basis of the supplied family history. Six types of tiles are available for students to use and 
there are a total of twenty positions to be filled in the pedigree (see Figure 1). If a student places a tile correctly 
in the pedigree it remains in its position and the name of the individual is displayed whereas incorrectly placed 
tiles are rejected and returned to their original position. 
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Figure 1: The interactive pedigree task illustrating the six types of tiles and a partially completed pedigree. 

We embedded a selection of audit trail functions into the Family Histories and Pedigrees section of the Medical 
Genetix courseware, placing particular emphasis on the interactive pedigree task. The audit included which 
screens within the section were visited and, for the interactive pedigree task, the amount of time students spent 
on the task, a detailed record of which tiles were dragged, where and in what sequence they were placed, and the 
number of times students consulted the history and help screens. 



Results 

A total of 78 students accessed Medical Genetix during the week it was designated as a resource for the medical 
curriculum. Of these, 49 entered the Family History and Pedigree section; indicating that 29 students bypassed 
this section altogether. Of those 49 students, 42 arrived at the interactive pedigree. Eight of the 42 students who 
arrived at the interactive pedigree did not attempt the task. The remaining 34 students, who attempted the task, 
form the basis of the analyses below. 



Activity 


Mean 


Minimum 


Maximum 


Total time on interactive pedigree* 


257.86" 


162.30 


576.40 


Number of correct drags 


15.62 


11 


20 


Number of incorrect drags 


5.91 


1 


13 


Number of incomplete drags 


.09 


0 


1 


Total time on history* 


125.46 


37.50 


322.00 


Average time on each history* 


8.63 


3.39 


22.95 


Number of times accessed history 


15.44 


5 


27 


Total time on help 


8.11 


2.80 


24.00 


Average time on each help 


7.58 


2.80 


24.00 


Number of times accessed help 


.21 


0 


2 



f two cases of the 34 were not included in this analysis because of incomplete data. 
A all times are recorded in seconds. 



Table 1: Summary statistics of students' use of the Interactive Pedigree derived from audit trail data. 
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Summary statistics of students’ activities while completing the interactive pedigree are presented in Table 1. On 
average, students spent around four and a half minutes on the task, made a little over fifteen correct tile 
placements and made around six mistakes while attempting the task. They spent a little over two minutes on the 
history screen and most accessed it regularly (over 85% of the sample accessed it more than ten times). 
However, students did not access the help screen regularly or for any great length of time. Eighty-two percent of 
^students did not access the help at all. 

We were particularly interested in determining how students were attempting the interactive pedigree task. As a 
result we focussed our attention on the number of correct and incorrect tile placements (or “drags”) that students 
completed. Table 2 presents frequency counts and percentages of students’ correct and incorrect drags. It can be 
seen from this table that all students completed over half the task (ie. completed over 10 drags correctly) but 
only one student successfully completed the entire task (ie. completed 20 drags correctly). The average number 
of tiles correctly placed before students left the task was between 1 5 and 1 6 and this seems to be a critical point 
in the task in terms of students interest or persistence (see Table 1). 



Correct Drags 


Freq 


% 


Incorrect Drags 


Freq 


% 


11 


1 


2.9 


1 


2 


5.9 


12 


1 


2.9 


2 


2 


5.9 


13 


3 


8.8 


3 


6 


17.6 


14 


4 


11.8 


4 


6 


17.6 


15 


8 


23.5 


5 


3 


8.8 


16 


6 


17.6 


6 


2 


5.9 


17 


5 


14.7 


7 


2 


5.9 


18 


3 


8.8 


8 


3 


8.8 


19 


2 


5.9 


9 


3 


8.8 


20 


1 


2.9 


10 


1 


2.9 


Total 


34 


100 


11 


0 


0 



12 


1 


2.9 


13 


3 


8.8 


Total 


34 


100 



Table 2: Number of correct and incorrect tile placements (“drags”) made by students 

The fact that only one student fully completed the interactive pedigree task raises questions as to why the 
students were not completing the entire exercise. Was the task too difficult or did the students simply lose 
interest? We attempted to address this question by considering the proportion of correct tile placements made by 
students. The number of correct tile placements made by students was divided by the total number of tiles placed 
by students (ie correct and incorrect drags). The results of this analysis (Table 3) reveal that students had a high 
degree of success with their tile placements with two thirds of students returning a 70% or better success rate. 



Proportion of Correct Drags 


Freq 


% 


< 50% 


1 


2.9 


51 -60% 


3 


8.8 


61-70% 


7 


20.6 


71-80% 


11 


32.4 


81-90% 


10 


29.4 


91 - 100% 


2 


5.9 


Total 


34 


100 



Table 3: Proportion of correct tile placements made by students. 

In addition to overall success rate, we analysed the final three tiles placed by students in the pedigree (regardless 
of how many tiles they placed in total). By assessing whether students were leaving the task after correctly or 
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incorrectly placing tiles we sought an indication of whether students were losing interest in the task or found the 
task too difficult. An analysis of the last three tile placements is presented in Table 4. 

These results indicate the overwhelming majority of students were leaving the task after performing well. Not 
one student placed all of the last three tiles incorrectly and 26 (76.5%) students placed their last three tiles 
correctly. Only one student left the task with an incorrect final drag, indicating that 33 (97.1%) students left the 
task with a correct final drag. This suggests that it was not the difficulty of the task that prevented students from 
completing it. Rather, given that students were successfully progressing within the task, their failure to complete 
it suggests that after a certain time they lost interest or had understood the point of the exercise. Further 
assessment is required to test this hypothesis further. 



Status of the last three tiles placed by students 


Freq 


% 


0 of last 3 tiles correct (0/3) 


0 


0 


1 of last 3 tiles correct (1/3) 


1 


2.9 


2 of last 3 tiles correct (2/3) 


7 


20.6 


3 of last 3 tiles correct (3/3) 


26 


76.5 


Total 


34 


100 



Table 4: Analysis of the last three tiles placed by students 



Discussion and Conclusions 

The results of this pilot study demonstrate the utility of audit trails for gathering evaluation data for the purposes 
of usage audits and formative evaluation of instructional design. Audit trails provided valuable information on 
the degree to which Medical Genetix was being used by students as a resource to investigate the problem of the 
week (78 of 184 students (42.4%) used the courseware over one week) as well as detailed information about 
which sections of the courseware were being. The audit trail system is clearly able to provide course 
coordinators with critical and accurate information about the integration of courseware into their curriculum. 
This is particularly pertinent in student-centred curricula such as the medical degree at the University of 
Melbourne, which emphasises self-directed learning. Common evaluation techniques, such as self-report 
questionnaires and observation, struggle to adequately gather information on the resources students use as part of 
their self-directed learning due to inherent variation in the times and locations that students access resources. 
Using the audit trail system we developed, data can be gathered easily while maintaining flexibility and freedom 
of student access. 

The audit trail system was used to formatively evaluate a specific interactive task (an interactive pedigree), and 
provided useful descriptive information. For example, of those students who attempted the task, all relied 
heavily on the provided history to complete the task but there was substantial variation in both the amount of 
time individual students spent on the history and the number of times they accessed it. The fact that the majority 
of students (82.4%) did not access a task specific help screen suggests the task was intuitive and well designed. 

Undoubtedly, the most interesting result from this component of the study was that all but one student failed to 
complete the interactive pedigree task. This, coupled with evidence that students were, in general, successfully 
progressing within the task indicates the task was not too difficult. There are a number of possible explanations 
for this finding. Students may have simply found the task too long and lost interest in the exercise. Alternatively, 
they may have understood the "point” or message of the exercise part way through the task, removing the 
incentive to complete it. It is also possible that students had completed the pedigree on a previous occasion when 
using Medical Genetix as a resource in a earlier problem of the week. 

The findings with regards to the interactive pedigree have implications for the instructional design of Medical 
Genetix as well as interactive tasks more generally. They suggest that attention should be given to length of 
interactive tasks and whether they are able to sustain students’ interest. This is especially relevant if students are 
directed to different parts of the courseware on the basis of whether they complete a task or not. While decisions 
about the instructional design of interactive tasks should always be made with the target audience in mind 
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(students’ previous experience and prior knowledge), it is clear that audit trail data can inform the design of 
these tasks. 

The success of this pilot implementation of the audit trail system validates our development of a generic yet 
flexible method of creating audit trails. By restricting the amount of information gathered using the systems’ 
inbuilt functions (see Judd & Kennedy, 2001) and by focussing on a specific interactive task, we were not 
overwhelmed by data and were able to gather meaningful and useful information. Care should be taken not to 
over-interpret the data garnered from audit trails and additional data collection techniques should be used in 
conjunction with audit data to assist in the analysis and interpretation of results. This not withstanding, the 
judicious implementation of the audit trail component of our electronic evaluation system clearly has the 
capacity to inform and assist both developers and evaluators. The two further roles of audit trails outlined by 
Misanchuk and Schwier (1992) (research in instructional design and counselling and advising) are also possible 
with the audit trail system we have developed (Judd & Kennedy, 2001) and will be investigated in subsequent 
implementations of the system. 
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